Performing actions associated with task items that represent tasks to perform

ABSTRACT

Techniques for processing task items are provided. A task item is electronic data that represents a task to be performed, whether manually or automatically. A task item includes one or more details about its corresponding task, such as a description of the task and a location of the task. Specifically, techniques for generating task items, organizing task items, triggering notifications of task items, and consuming task items are described. In one approach, a task item is generated based on input from a user and context of the input. In another approach, different attributes of task items are used to organize the task items intelligently into multiple lists. In another approach, one or more criteria, such as location, are used to determine when to notify a user of a task. In another approach, actions other than generating notifications are enabled or automatically performed, actions such as emailing, calling, and searching.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority to U.S. Provisional PatentApplication No. 61/493,201, entitled “Generating And Processing DataItems That Represent Tasks To Perform,” filed on Jun. 3, 2011, inventedby Thomas R. Gruber, et al., the entire disclosure of which isincorporated by reference for all purposes as if fully set forth herein.

This application is related to U.S. patent application Ser. No.12/479,477, filed Jun. 5, 2009, the entire contents of which are herebyincorporated by reference as if fully set forth herein.

This application is related to U.S. patent application Ser. No.12/987,982, filed Jan. 10, 2011, the entire contents of which are herebyincorporated by reference as if fully set forth herein.

FIELD OF THE INVENTION

The present invention relates to electronic reminders and, moreparticularly to, the intelligent generation, organization, triggering,and delivery of reminders and tasks in electronic to-do lists.

BACKGROUND

People have devised numerous ways to remind themselves of certain tasksor events. Many people have and still do write on physical media, suchas sticky notes and calendars. With the ubiquity of electronic devices,many people have turned to computers to help manage their to-do listsand keep of record of upcoming events. Numerous reminder and to-doapplications are available, both for desktop computers as well ashandheld devices, such as laptop computers, tablet computers, and“smart” phones.

However, the timeliness and accuracy of a notification provided to auser of a reminder application depends almost entirely on input receivedfrom the user. For example, if a user enters, in a reminder application,a wrong date for an important event, then the user might not receive anotification of the event until after the event occurs. As anotherexample, if a user provides a generic description of a task (e.g., “sendhim an email”) in a to-do application, then, when the user later readsthe description, the user might not remember who “him” is and/or whatthe content of the email should be. In other words, when it comes toreminder and to-do applications, the old adage of “garbage in garbageout” is applicable.

The approaches described in this section are approaches that could bepursued, but not necessarily approaches that have been previouslyconceived or pursued. Therefore, unless otherwise indicated, it shouldnot be assumed that any of the approaches described in this sectionqualify as prior art merely by virtue of their inclusion in thissection.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings:

FIG. 1 is a block diagram that depicts a system that is used forcreating and processing task items, according to an embodiment of theinvention;

FIG. 2 is a flow diagram that depicts a process for generating a taskitem based on context of user input, according to an embodiment of theinvention;

FIG. 3 is a flow diagram that depicts a process for determining a timeto provide a reminder to a user ahead of a scheduled time for a task,according to an embodiment of the invention;

FIG. 4 is a view of a travel reminder, according to an embodiment of theinvention;

FIGS. 5-15 depict views of various types of lists, according to anembodiment of the invention; and

FIG. 16 is a block diagram that illustrates a computer system upon whichan embodiment of the invention may be implemented.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however,that the present invention may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order to avoid unnecessarily obscuring thepresent invention.

General Overview

Multiple techniques are provided below for assisting a user in managinghis/her tasks. Tasks are represented as task items in a computingsystem. Specifically, approaches for generating task items, organizingtask items, triggering the notification of tasks based on correspondingtask items, and consuming task items are described in detail below.

With respect to generating a task item, a task item is generated basedon input from a user. The input (whether voice or text) includes one ormore details about a task while context of the input is used todetermine one or more other details about the task. The one or moreother details are not reflected in the input. Context may include datathat is stored on a device of the user. For example, a user provides thefollowing voice input: “Call George at 5 PM today.” The user may have acontact list that includes information about multiple people with thefirst name of George. However, based on the context of the input, suchas a recent voice message from a phone number associated with aparticular George, it can be determined which George the user intends tocall. In this way, a user is not required to be specific about eachdetail of a task when providing input about the task.

With respect to triggering a notification of a task item, one or morecharacteristics of a device may be used to determine whether to triggerthe notification. Thus, time is not the sole factor (if at all) ofwhether a notification should be provided to a user. Examples of suchcharacteristics may include where the device is located, what the deviceis displaying or processing, and specific movements of the device. Forexample, the fact that the device is in a car or at work may trigger thegeneration of a reminder of a task. As another example, if the device iscurrently displaying web content, then a user of the device may beconsidered to be “online,” which status might trigger a notification ofa task to be provided to the user.

With respect to “consuming” task items, instead of simply providing areminder of a task, the task is automated so that a user is not requiredto perform the steps typically required to perform the task. Forexample, a user may want to call a particular person at a particulartime. When the particular time equals the current time, instead ofsimply reminding the user about the call, the call can be set up, readyto make without the user having to specify the person's phone number.

With respect to organizing task items, task items may be organizedautomatically or manually into multiple lists. Each list corresponds toa different attribute of a task item, such as the type of task, the typeof criteria that is used to trigger a notification of a task, and thelocation of where the task is to be performed.

Task Items

Again, a “task item” is an electronic data item that contains one ormore details about a task to perform, whether by a user or automaticallyby a process. A task item is generated based on input from a user. Atask item may be one of two types: tasks associated with reminders(“reminder task”) and tasks not associated with reminders (“non-remindertask”). A reminder task is associated with an action, such as anotification being generated and provided to a user, while anon-reminder task is not associated with any action. A non-reminder taskmay be associated with a “complete-by” date or time. However, thecomplete-by date or time does not trigger the creation of a notificationor other action. In other words, while a reminder task is associatedwith one or more triggering criteria that, when satisfied, trigger anaction, a non-reminder task is not. Thus, a “task item” may or may notbe associated with one or more triggering criteria that, when satisfied,triggers an action.

System Overview

FIG. 1 is a block diagram that depicts a system 100 that is used forcreating task items and processing task items, according to anembodiment of the invention. System 100 includes a device 110, a network120, and a cloud 130.

Device 110 is any computing device that is capable of receiving inputfrom a user and displaying information about tasks. Non-limitingexamples of device 110 include a desktop computer and a handheld device,such as a laptop computer, a tablet computer, and a “smart” phone. Inthe illustrated embodiment, device 110 includes a task manager 112. Taskmanager 112 processes task items, both of the reminder type or of thenon-reminder type. Task manager 112 may be implemented in software,hardware, or any combination of software and hardware.

Device 110 includes communication technology (e.g., wireless technology)for sharing information with other devices. Device 110 can include avariety of integrated user interface units or can be coupled to userinterface units through one or more communication ports or data links ofthe device. Non-limiting examples of user interface units include avoice input unit (e.g., a microphone), physical input units (e.g., akeyboard, a mouse, a track ball, a rotary dial or wheel, a touchpad, ora touch-screen), and motion sensors (e.g., an accelerometer,magnetometer, or a gyroscope). Any of these user interface units can beimplemented as an external unit that communicates with device 110 toprovide user input using a wired or wireless communication technology.Examples of wired communication technology include Universal Serial Bus(USB) interface, FireWire interface, etc. Examples of wirelesscommunication technology include Bluetooth, Wi-Fi, and WiMax, infrared.Through these user interface units, device 110 can receive physical orvoice inputs from the user.

Device 110 includes one or more output units to present visual and audioinformation to a user. Non-limiting examples of output units include adisplay unit for displaying visual data and a speaker for playing audio.

Cloud 130 is implemented by one or more computing devices. Cloud 130hosts multiple services, such as a NLP (natural language processing)service 132 and one or more other services 134A-N. NLP service 132 usesone or more models of real-world things that a user can talk about inorder to make sense of what the user is trying to say. For example, NLPservice 132 can determine, based on models and context, what a user maybe referring to when the user uses terms like, “him,” “there,” or“that.” An example of how NLP service 132 might operate is described inU.S. patent application Ser. No. 12/987,982, referenced above.

NLP service 132 may employ numerous APIs to communicate with andinitiate actions performed by the one or more other services 134A-N and,optionally, other services not hosted in cloud 130. For example, inresponse to voice data sent from device 110, where the voice datareflects the user command “Reserve two seats at Maribella's in San Joseat 7 PM tonight,” NLP service 132 makes an API call to an onlinereservation service provided by Maribella's restaurant to initiate thecreation of two reservations at that restaurant for 7 PM. Thus, NLPservice 132 allows many operations to be performed automatically withoutrequiring a user of device 110 to manually input text data and interactwith numerous applications.

Communication between device 110 and services hosted in cloud 130 ismade possible via network 120. Network 120 may be implemented by anymedium or mechanism that provides for the exchange of data betweenvarious computing devices. Examples of such a network include, withoutlimitation, a network such as a Local Area Network (LAN), Wide AreaNetwork (WAN), Ethernet or the Internet, or one or more terrestrial,satellite, or wireless links. The network may include a combination ofnetworks such as those described. Without limitation, the network maytransmit data according to Transmission Control Protocol (TCP), UserDatagram Protocol (UDP), and/or Internet Protocol (IP).

The following description includes numerous examples where both device110 and cloud 130 take part in generating task items, organizing taskitems, triggering notifications of task items, and consuming task items.Instead, one or more of the techniques described herein may beimplemented wholly on device 110 (making network 120 and cloud 130unnecessary, wholly in cloud 130, or using some combination of device110 and cloud 130.

Processing of Task Items

Task items may be created on device 110 or in cloud 130 based on inputreceived at device 110. Although not depicted, task items may be storedon device 110 or in cloud 130, or synchronized to both. If task itemsare stored in cloud 130, then task manager 112 may retrieve the taskitems in response to, for example, input from a user or the one or moretriggering criteria associated with one or more task items beingsatisfied.

In the scenario where task items are created and stored in cloud 130,task manager 112 may be, primarily, a set of one or more user interfacesthat display information about tasks. Thus, a task service (not shown)in cloud 130 would be responsible for maintaining task items andtriggering any notifications when triggering events occur.

Alternatively, task manager 112 creates and stores task items on device110. In this scenario, task manager 112 may be entirely responsible formaintaining task items and generating any notifications when triggeringevents occur. One advantage of this scenario is that device 110 may beoperating in an “offline” mode where device 110 is not capable ofcommunicating with any service hosted in cloud 130.

Further, in this scenario, device 110 may include a service like NLPservice 132, which may be part of task manager 112 or may executeseparately from task manager 112. Such a service acts as aconversational interface to allow a user to quickly and easily createtasks. Such a service may be implemented by a process that iscontinually executing in the background without requiring a user ofdevice 110 to provide input to cause the service to execute. Thus,whenever device 110 starts up (or restarts), the service isautomatically started.

Alternatively, information needed to create task items may be identifiedby NLP service 132 (i.e., hosted in cloud 130). Device 110 may include auser input interface that continuously executes in the background,identifies input (e.g., voice or text) from a user, and sends the inputover network 120 to NLP service 132. Once NLP service 132 identifiestask details in the input, NLP service 132 may send task information(e.g., a description of a task and a time to complete the task) (a) overnetwork 120 to task manager 112, which creates and stores a task itembased on the information or (b) to a task service in cloud 130 to createa task item based on the information.

Most of the examples provided herein involve NLP service 132 receivinginput data from device 110, identifying details (about a task) reflectedin the input data, and providing those details to task manager 112.However, embodiments of the invention are not limited to this scenario.Such examples may alternatively involve only device 110 or may involvedevice 110 as merely an input and display device where NLP service 132and a task service in cloud 130 provide the primary functionality.

I. Generating Task Items Based on Context

According to an embodiment of the invention, a task item is generatedbased on input and context of the input. “Context” of input refers todata that is currently or recently (relative to input, from a user, thatinitiated the generation of a task item) displayed or processed atdevice 110. Thus, context data is not reflected in the input from theuser. For example, a user of device 110 may provide the following voiceinput: “Send him an email about the project when I get home.” Thepronoun “him” is ambiguous because it is not clear, from the inputalone, to whom “him” refers. However, the context of the voice input maybe that device 110 currently displays (or just recently displayed) anemail from an individual named Jack Bauer where the email includes arequest for a status update about a project named “Bunny.” Based on thevoice input and the context, task manager 112 (or a task service incloud 130) creates a task item that includes the description “Send JackBauer an email about Project Bunny” and that includes the triggeringcriterion of device 110 being at a geographical location that is at ornear the user's home. When device 110 is at or near the user's home,task manager 112 causes a notification to be displayed on device 110where the notification includes the description from the task item.

FIG. 2 is a flow diagram that depicts a process 200 for generating atask item based on context of user input, according to an embodiment ofthe invention. At step 210, input that expressly specifies one or morefirst attributes for a task is received from a user. The input may betext input or voice input. The text input may be from a user of device110 pressing physical keys on device 110 or pressing a touch screen ofdevice 110 that includes a graphical keyboard. Additionally oralternatively, device 110 includes a microphone that accepts, from auser, voice input that device 110 converts into voice data. Device 110may send the input data (whether voice data or text data) to NLP service132, which analyzes the input data to identify the one or more firstattributes for the task. Instead, as noted previously, device 110 mayinclude functionality to analyze the input data to identify the one ormore first attributes for the task. (Although many of the examplesherein refer to natural language processing, natural language processingis not required.)

At step 220, a task item is generated for the task based on the inputdata. At step 230, one or more second attributes for the task aredetermined based on context data that is separate from the input.Although step 230 is depicted as occurring after step 220, step 230 mayoccur before step 220.

At step 240, the one or more first attributes and the one or more secondattributes are stored in association with the task item.

The steps of process 200 may be performed by one or multiple devices.For example, the input in step 210 may be processed at device 110 togenerate the task item. In this scenario, task manager 112 (or anotherprocess executing on device 110) identifies the context data associatedwith the input to determine the one or more second attributes, for thetask, that are not identified in the input. Task manager 112 then storesthe one or more second attributes in or in association with the taskitem.

Alternatively, in another scenario, device 110 sends the user input overnetwork 120 to NLP service 132. NLP service 132 accepts, as input,context data associated with the input to determine the one or moresecond attributes, for the task, that are not identified in the input.Context data may have been sent to NLP service 132 prior to the inputthat initiates the generation of the task item (in step 220). NLPservice 132 sends the one or more second attributes to task manager 112(or a task service in cloud 130). Task manager 112 stores the one ormore second attributes in or in association with a newly-generated taskitem.

Certain words or phrases may be used to cue NLP service 132 tocommunicate with manager 112. For example, user commands that begin with“Remind me . . . ” and “I need to . . . ” are used by NLP service 132 todetermine to communicate with task manager 112. In response to detectingone of those user commands, NLP service 132 analyzes the input data(from device 110) and, optionally, context data for certain types oftask details, such as a location, time, description, and/or action. NLPservice 132 then determines to communicate with task manager 112 andsends, to task manager 112, the task details as part of thecommunication(s).

Sources of Context Data

Context data associated with user input that initiates the generation ofa task item may come from one of many sources. Non-limiting examples ofcontext data include data that is or was displayed on device 110(“display data”), data that is stored on or in association with device110 (“personalized data”), data that is or was processed by device 110(“process data”), data that was previously provided by a user of device110 (“input data”), data that indicates the location of device 110(“location data”).

The following is an example of display data, or data that is or wasdisplayed on device 110. Device 110 displays a map that includes amarker associated with a specific location on the map. A user of device110 then says, while the map is displayed or soon after the map wasdisplayed, “I need to be there by 5 today.” NLP service 132 (or a voiceanalyzer on device 110) analyzes voice data that reflects the voiceinput. NLP service 132 analyzes data that is currently displayed ondevice 110 to determine what “there” refers to. NLP service 132identifies the marker and the associated location and replaces “there”with the location. NLP service 132 sends, to task manager 112, task datathat indicates 5 PM today as the completion time of the task and thespecified location as the location of the task. Task manager 112generates a task item based on the task data.

As another example of display data, device 110 displays an email that isfrom a particular sender and includes a subject line. A user of device110 then says, “I need to email him about that subject in two hours.”Device 110 sends voice data that reflects this input and an image ofwhat is displayed to NLP service 132. In response, NLP service 132identifies the email address of the sender of the email and the subjectof the email. NLP service 132 sends, to task manager 112, task data thatindicates a time of two hours from the current time as the completiontime of the task and

The following is an example of personalized data, or data that is storedon or in association with device 110. A user of device 110 says, “I willhave lunch with Rachelle tomorrow at 12 noon.” Device 110 sends voicedata that reflects this input to NLP service 132, which identifies“Rachelle” in the voice data. NLP service 132 looks up “Rachelle” incontact data or an “address book” (stored on device 110 or in cloud 130)and determines that the last name of Rachelle is Goodwin. NLP service132 then causes “Rachelle Goodwin” to be associated with a task itemthat is generated for the task. In addition to or instead of beingstored on device 110, personalized data may be stored in cloud 130,i.e., remote to device 110.

The following is an example of process data, or data that was recentlyprocessed by device 110. For example, a user of device 110 used device110 as a phone to communicate with a friend. Device 110 keeps track ofwho the user recently spoke with. After ending the call, the user says,“Remind me to call her back in 30 minutes.” NLP service 132, in additionto analyzing the voice input, analyzes data that indicates who recentlyestablished a phone call with device 110 (e.g., the last five phonecalls). NLP service 132 determines the phone number of the most recentlyestablished phone call with device 110. NLP service 132 then determines,based on contact data, that the phone number is associated withparticular individual. NLP service 132 sends, to task manager 112, taskdata that indicates a task of calling, a time of 30 minutes from thecurrent time as the completion time of the task, the name of theparticular individual, and, optionally, the phone number of theparticular individual. Task manager 112 generates a task item based onthe task item.

The following is an example of input data, or data that was recently(e.g., the last 5 minutes) provided by a user of device 110. The inputfrom the user may be text input or voice input. Device 110 or NLPservice 132 keeps track of recently entered input and may use that inputto determine the identity of certain terms reflected in current input.For example, a user of device 110 says, “Remind me to meet him there at7 tonight.” NLP service 132 receives voice data that reflects that voiceinput and identifies the terms “him” and “there.” Although it is notclear who “him” is and where “there” is, NLP service 132 accesses inputthat was recently received from the user. Such recently-received inputreflects the names “George Reed” (identified as a name of a person) and“Starbucks” (identified as a place). In response, NLP service 132 causesa task item to be generated where the task is to “Meet George Reed atStarbucks” where the time is 7 PM of the current day.

The following is example of location data, or data that indicates alocation of device 110, whether current or past. A user of device 110says, “Remind me to meet Julian here next Thursday for lunch.” Device110 sends voice data that reflects this input to NLP service 132. NLPservice 132 identifies the term “here” and, in response, determineswhere device 110 is currently located. The current location may bedetermined in numerous ways. For example, device 110 may provide, to NLPservice 132, a geographical location, such as longitude and latitudecoordinates. NLP service 132 may then determine, based on thecoordinates, a name of the place or establishment that is located atthose coordinates. NLP service 132 causes a name of the place orestablishment to be associated with a task item for the task to meetJulian for lunch on the date indicated.

Alternatively, the user may say, “I need to meet Josh Peters tomorrow atthe same place where I was last Thursday at noon.” Device 110 sendsvoice data that reflects this input to NLP service 132. NLP serviceidentifies the phrase “at the same place where I was last Thursday atnoon” and, in response, determines where device 110 was located lastThursday at noon. NLP service 132 accesses location history data (storedin cloud 130 or stored on device 110 and sent to NLP service 132) anddetermines where device 110 was located last Thursday at noon. Thelocation history may indicate the name of a place or may consist ofgeographical coordinates. If geographical coordinates, then NLP service132 determines a name of the place or establishment that is located atthose coordinates. NLP service 132 causes that name to be associatedwith a task item for the task to meet Josh Peters on the date indicated.

Events that occur with respect to device 110 may also be used to createtask items. Such events may fall into one or more categories (or types)of context data described above, such as display data, presentationdata, and process data. For example, device 110 detects an incoming calland notifies the user of the call by causing a phone number or otheridentifying information about the call or caller to be displayed on ascreen of device 110. In addition to this information, the display mayinclude three selectable options: “Answer”, “Ignore”, and “Call BackLater.” If the user selects “Call Back Later”, then a task item iscreated where the task item identifies the caller and, optionally, atime of the call and/or a time to make a call to the caller. Also, thetask item may be automatically categorized as a task of type “To Call.”

Many of the examples herein regarding generating task items include auser providing voice or text input that includes details about a task.Another non-limiting example of how a task item may be generated is auser selecting (or highlighting) text that is displayed on a screen ofdevice 110. The selected text is considered context data. After the textis selected, the user may be presented with one or more options, one ofwhich is a “Remind” option which, when selected, causes a task item tobe generated. Task manager 112 generates the task item based on theinformation reflected in the selected text. Details of the task item maybe also determined from other context data, such as a time or event totrigger a notification of the task.

Virtual Dialogue

In some situations, NLP service 132 is unable to determine one or moredetails about a task based on input received from device 110 and thecontext associated with the input. Thus, in an embodiment, NLP service132 prompts a user of device 110 for further input to determine the oneor more details. The one or more details may pertain to any attribute ofa task item, such as the description of the task, the location of thetask, the location of a reminder (if any), or the time of the task.

For example, NLP service 132 receives, from device 110, voice data thatreflects a user's command to “Remind me to call Steve at 7.” NLP service132 may have access to information (e.g., an address book) aboutnumerous contacts, of the user, that have the name of Steve. Further,nothing in the address book can be used to disambiguate which of theSteve contacts to call. Therefore, NLP service 132 sends, to device 110,the following message to be displayed (or played audibly) by device 110:“Do you mean Steve Anderson, Steve Hanson, or Steve Jobs?” The user thenprovides, to device 110, voice or text input that indicates one of thethree Steve contacts. In response, device 110 sends the correspondingvoice or text data over network 120 to NLP service 132.

As another example, NLP service 132 receives, from device 110, voicedata that reflects a user's command to “I need to pick up bread at WholeFoods.” In response, NLP service 132 performs a lookup of the nearestWhole Foods stores to (a) the current location of device 110 or (b) theuser's home. There may be multiple Whole Foods stores that are neardevice 110's current location and near the user's home. Therefore, NLPservice 132 sends, to device 110, the following message to be displayedby device 110: “Which Whole Food's? The one on Almaden Rd, ChesterExpressway, or Green Street?” The user then provides, to device 110,voice or text input that indicates one of the three Whole Foods stores.In response, device 110 sends the corresponding voice or text data overnetwork 120 to NLP service 132.

As another example, NLP service 132 receives, from device 110, voicedata that reflects a user's command to “Remind me to text Jerry by 8.”In response, NLP service 132 determines, based on the voice data and thecontext of the input that Jerry is Jerry Wall, indicated in the user'scontact list (or address book). However, it is unclear whether the userintended 8 AM or 8 PM as the time to send an SMS message to Jerry.Therefore, NLP service 132 sends, to device 110, the following messageto be displayed by device 110: “Do you want to text Jerry Wall at 8 AMor 8 PM?” The user then provides, to device 110, voice or text inputthat selects one of the two times. In response, device 110 sends thecorresponding voice or text data over network 120 to NLP service 132.

Autocategorization of Task Items

In an embodiment, NLP service 132 determines, based on input from a userof device 110, one or more categories to associate with a task item. Theone or more categories may be one of many different categories, whichmay be virtually limitless. Non-limiting examples of categories withwhich a task item may be associated include things to purchase, thingsto do on vacation, things to do at work, and things to do while driving.Each category may be associated with a sub-category. For example, a“purchase category” may be divided into a grocery category indicatingitems to purchase at a grocery store, a book category indicating booksto purchase, and a music category indicating songs to purchase.

For example, a user may provide the following voice input to device 110:“Remind me to get milk.” Device 110 sends voice data that reflects thatinput to NLP service 132. NLP service 132 determines that a task itemshould be created and that “get milk” should be the descriptionassociated with the task item. NLP service 132 may also determine thatmilk is a grocery item and that the task item should be associated witha grocery category and/or a purchase category. Thus, NLP service 132 maysend, to task manager 112, category data that indicates one or morecategories with which the task item (whether created by NLP service 132,by a task service in cloud 130, or by task manager 112) should beassociated.

As will be described hereinafter, the one or more categories associatedwith each task item may be used to organize task items that belong tothe same category and display, on device 110, task items of the samecategory. This will allow a user of device 110 to view task items bycategory, in addition to or instead of by completion time, by creationtime, by trigger type (described hereinafter), by location, by type(e.g., reminder task v. non-reminder task), or by some other criterion.

II. Triggering Notifications of Task Items

As noted previously, a task item may be associated with one or moretriggering criteria (or triggers) that, when satisfied, causes anotification to be presented to a user of device 110 or some otheraction to be performed. When one or more triggering criteria of a taskitem are satisfied, a notification (or other action) is “triggered.”Non-limiting examples of triggering criteria include time, location,relative travel time, context triggers, and exogenous triggers, each ofwhich is described in more detail below.

Time Trigger

The time of a time trigger may be an absolute time, a relative time, arecurring time, or a symbolic deadline. An example of an absolute timeis Jun. 6, 2011, 9 AM Pacific Time. An example of a relative time is “10minutes before the Patriots-Jets football game.” An example of arecurring time is “Every Thursday at LOAM.” An example of a symbolicdeadline is “end of business day”.

Location Trigger

According to an embodiment of the invention, the location of device 110is a triggering criterion associated with a task item. Such a triggeringcriterion is referred to herein as a “location trigger.” The location ofdevice 110 may be determined in one of many ways. For example, thelocation of device 110 may be automatically determined based on Wi-Fipositioning, cell positioning, and/or GPS (global positioning system)positioning. Device 110 may determine its current location with orwithout input from a service in cloud 130.

In an embodiment, a user may provide input that indicates a label to beassociated with a certain geographical location. For example, a user ofdevice 110 may speak the following sentence, “I am home” or “I am atWhole Foods.” NLP service 132 may then associate the word “home” orphrase “Whole Foods” with the current location of device 110, asdetermined based on one of the three positioning methods mentionedpreviously. This association of a word with a location may be laterleveraged to determine where “home” or “Whole Foods” is located.

A location trigger may not be associated with a specific geographiclocation or area. Instead, a location trigger may be associated with aplace that is not limited to a specific geographic location or area. Forexample, a location trigger of a task item may be “on the road” or“while driving.” Device 110 (or a process executing on device 110)determines that the current location of device 110 is on a freeway oranother busy road. Thus, this determination can be made regardless ofthe speed at which device 110 is moving or whether device 110 is pairedwith another device that would indicate that the user is traveling.Based on this determination, task manager 112 analyzes one or more taskitems to determine whether any task items are associated with the “onthe road” or “while driving” location trigger.

As another example, a location trigger of a task item may be the user'scar. Specifically, the user may have provided the following voicecommand: “Remind me to call my mom while driving.” NLP service 132analyzes voice data that reflects that command and determines that“while driving” refers to the user's car. The user's car may have aBluetooth-enabled component to allow device 110 to communicate with theuser's car. When device 110 comes into range of a Bluetooth signalpropagated by a Bluetooth-enabled component in the user's car, device110 determines that device 110 is located in (or at least near) theuser's car. In response to this determination, task manager 112 triggersthe location trigger of the task item. Task manager 112 causes areminder message to be displayed on device 110, where the remindermessage informs the user to call his mother. The user may then provide asingle tap or a voice response that causes a phone application executingon device 110 to initiate a call to a phone number associated with theuser's mom.

While establishing a connection (or “pairing”) with anotherBluetooth-enabled device is one example of pairing that can be used todetermine device 110's location, other types of pairings are possible.For example, device 110 may detect certain network data during theevening and morning hours. The network data indicates one or morenetworks to which device 110 may connect. The network data may includethe names of one or more networks or MAC addresses of one or morerouters. Device 110 may then determine that whenever that network datais detected, device 110 is considered to be at the user's home. Thus,actual pairing is not required since pairing entails the establishmentof a connection between device 110 and another device, such as a router.As another example, device 110 may detect a Wi-Fi signal on a train,subway, or bus. The Wi-Fi signal might indicate the type oftransportation that corresponds to the Wi-Fi signal. Thus, device 110might detect, based on the Wi-Fi signal, that its location is “on atrain,” “in a subway,” or “on a bus.” If a triggering criterion of atask item indicates one or more of these locations, then an actionassociated with the task item may be triggered. Further, such“transit-oriented” locations may also be considered to be associatedwith specific contexts (described in more detail below), such as “intransit” or “while traveling.” Thus, detection by task manager 112 ofsuch contexts may cause actions associated with certain task items to beperformed.

The foregoing examples of location triggers can be categorized as“arrival triggers,” such as are found in user input to “Remind me to doX when I arrive at Y.” Another type of location trigger is a “departuretrigger,” an example of which is found in the user command to “Remind meto do X when I leave work” or “ . . . when I leave here.” In anembodiment, in the departure trigger scenario, a minimum distance fromthe current location and the location of the departure is requiredbefore a particular departure trigger “fires.” Such a minimum distancemay be helpful to avoid the performance of corresponding actions whenthere are false starts.

Additionally, a location trigger may be one of multiple conditions thattrigger an action of a task item. Examples of user commands that includemultiple conditions include “Remind me to do X when I get home or at 8PM at the latest,” “Remind me to do X before 8 PM or when I leave,whichever is first,” and “Remind me to do X before 8 PM or while I amdriving, whichever is first.”

Travel Time Trigger

In an embodiment, the location of device 110 and a time associated witha task item is used to provide a notification to a user of device 110.Thus, while the time may be one of the one or more triggering criteriaassociated with the task item, the location of device 110 may not be, atleast explicitly so.

FIG. 3 is a flow diagram that depicts a process 300 for determining atime to provide a reminder to a user ahead of a scheduled time for atask, according to an embodiment of the invention. Process 300 may beperformed by one or more processes executing on device 110 or in cloud130. However, for ease of explanation, all the steps in process 300 areperformed by task manager 112.

At step 310, task manager 112 determines a current location of device110. At step 320, task manager 112 determines a location of adestination (or “destination location”) associated with (or identify by)a task item. At step 320, based on the distance between the twolocations, task manager 112 determines a “travel time,” or the time itmight take for the user of device 110 to travel to the destinationlocation. At step 330, task manager 112 determines a “difference time,”or the difference between the current time and the time triggeringcriterion associated with the task item. At step 340, if the travel timeis the same as or near the difference time, then task manager 112provides a notification to the user. This notification acts as areminder for the user to begin (if s/he has not already done so)traveling to the destination.

For example, a task item may be for a reminder to meet Sue at aparticular restaurant at 2 PM. Task manager 112 determines the locationof device 110 and the location of the particular restaurant. Thelocation of the particular restaurant may be determined by initiating,e.g., an Internet search and identifying the closest restaurant, withthe same name, to device 110's location. Alternatively, an address ofthe particular restaurant may already be stored in association with thetask item. Based on the distance between device 110's location and theparticular restaurant, task manager 112 determines how long it will takefor the user of device 110 to travel to the particular restaurant (or“travel time”). When the travel time is the same as or near (e.g.,within 10 minutes) the difference between the current time and the timetrigger (i.e., 2 PM), then task manager 112 causes, to be displayed ondevice 110, a message that indicates that the user should leave soon toarrive at the particular restaurant at 2 PM.

In an embodiment, the time of when to leave for a destination changesbased on the current location of device 110. For example, when thecurrent location of device 110 is at location A and the destination isat location B, task manager 112 determines that the user should begintraveling 50 minutes before the time of a scheduled task. However, inresponse to detecting that the current location of device 110 is now atlocation C, task manager 112 determines that the user should begintraveling 20 minutes before the time of the scheduled task. For example,a user of device 110 may be at home at the beginning of the day and taskmanager 112 determines that it will take 50 minutes to travel, from theuser's home, to the location of a dinner event in the evening. Later inthe day, the user of device 110 travels to work, which is closer to thelocation of the dinner event. In response to device 110 being at adifferent location, task manager 112 determines that it will take 20minutes to travel, from the user's work, to the location of the dinnerevent.

In an embodiment, the time of when to leave for a destination changesbased on current traffic information. For example, at 2:30 PM, taskmanager 112 determines that the time of when a user of device 110 shouldleave for a restaurant is 5:00 PM. However, due to a car accident on afreeway that the user can take to arrive at the restaurant, the trafficslows considerably. Task manager 112 determines, at 3:30 PM, that thetime of when the user should leave for the restaurant is 4:00 PM.

FIG. 4 is a view 400 of a travel reminder, according to an embodiment ofthe invention. View 400 is displayed by device 110. The travel reminderof view 400 contains six data items. The six data items include: (1) adescription 410 of the corresponding task (“pick up Chloe”); (2) a time420 of when to complete the task (“5:00 PM Today”); (3) an action 430 toperform when the user of device 110 should begin traveling to thedestination; (4) a reminder time 440 that indicates that the user wouldlike to be reminded of when the user should begin traveling to arrive atthe destination on time; (5) a start time 450 that indicates when theuser should begin traveling to arrive at the destination on time; and(6) a location 460 that indicates a name of the destination and anaddress of the destination. Another travel reminder that device 110displays may contain more or less data items.

The action associated with action 430 may be triggered (or performed) inresponse to task manager 112 determining that the current time(indicated at the top of travel reminder) equals the time indicated bystart time 450. In the illustrated example, action 430 is a map-relatedaction where task manager 112 causes a map to be generated at start time450 and displayed to the user of device 110. The map includes anindication of the address of location 460, an indication of the user'scurrent location, or both. Instead of automatically causing the map tobe displayed at start time 450, task manager 112 might first cause amessage to be displayed on device 110, wherein the message includes anoption to generate the map. If the user selects the option (e.g.,through voice input or tapping on the screen), then task manager 112causes the map to be generated and displayed.

A reminder setting may be in an “on” or “off” mode. In FIG. 4, remindertime 440 is in an “on” mode. If reminder time 440 is in an “off” mode,then the travel reminder of view 400 might not include reminder time 440or start time 450.

As indicated previously, task manager 112 might change start time 450 inresponse to changes in device 110's location. Thus, while start time 450may indicate “3:30 PM Today” when device 110 is located at the user'shome in the morning, start time 450 may indicate “4:20 PM Today” whendevice 110 is located at the user's work office in the afternoon.

In an embodiment, task manager 112 checks for changes in computed starttime 450 in response to significant changes in device 110's location.Significant changes in location may be determined as a result of otherevents that are already being computed. For example, device 110 mightalready process events when it transitions between cell towers, andthese events could trigger the re-computation of a change in locationand, therefore, in an updated start time 450. Other non-limitingexamples of events that indicate a potential significant change inlocation are changes in Wi-Fi signatures detected, the computation ofaccurate GPS locations for some other application (such as maps ornavigation), a power cycle event, turning on or off radios on thedevice, alerts based on accelerometer signals, and the receipt of textmessages or push notifications that contain location information.

In an embodiment, task manager 112 combines strategies for detectingsignificant event changes. For example, in a low power/low resolutionmode, task manager 112 only checks for significant location changesevery N minutes or only when some periodic computation occurs, such aschecking for incoming data. In a high power/high resolution mode, taskmanager 112 uses cell tower positioning and/or GPS. A combined strategymight run the low power solution by default and then invoke the highpower solution when the estimated start time is soon or when otherevents occur (for example, a change in Wi-Fi or Bluetooth signatures isdetected).

In an embodiment, a travel reminder or start time item in a travelreminder may be associated with one or more modes of transportation.Non-limiting examples of modes of transportation include driving a car,riding a bus, bicycling, and walking. A default transportation mode maybe driving a car. For example, task manager 112 may provide the optionfor a user to view start time 450 in a “car” mode, a “bus” mode, a“bike” mode, a “walking” mode, or multiple modes simultaneously.Depending on the current mode(s) selected for start time 450, the starttime may vary widely. For example, in FIG. 4, while start time 450indicates “4:20 PM Today” for a car mode, start time 450 may indicate“3:15 PM Today” for a bus mode, “3:45 PM Today” for a biking mode, and“11:30 AM Today” for a walking mode.

In a related embodiment, a task item is associated with both a locationand a date/time and a notification of the task may be triggered byeither the user (or, rather, the user's device) being at the location orby the date/time. For example, if the user's device is at the location,(either on the date or regardless of the date), then a notification istriggered. If the user has not arrived at the location on the dayindicated by the date (or at the location by the time), then the time isused as a “last resort” for triggering a notification.

Context Triggers

As described previously, time and location are examples of types oftriggering criteria associated with a task item. Another type oftriggering criteria associated with a task item is context. A “contexttrigger” refers to one or more characteristics of device 110 other thansimply the device 110's location. Thus, like context triggers, traveltriggers and travel time triggers also refer to one or morecharacteristics of device 110.

Context triggers may be categorized into one of two types: data-orientedcontext triggers and spatial-oriented context triggers. Non-limitingexamples of data-oriented context triggers include the kind or type ofdata that device 110 is displaying on its screen (e.g., video), thespecific application(s) or type of application(s) that are currentlyexecuting on device 110 (e.g., a texting application or a musicapplication), the type of input device 110 is receiving from a user(e.g., voice or data), and the type of network connections available todevice 110 (e.g., Wi-Fi or cellular network).

For example, a user command that device 110 receives may be “Remind meto call my mom next time I am on the phone.” The phrase “on the phone”is presumed to mean that when the user is using device 110 as a phone, areminder will be sent to the user to inform the user to call his/hermom.

As another example, a user command that device 110 receives may be “Ineed to email Bob when I am surfing the Internet.” The phrase “surfingthe Internet” is presumed to mean that when the user is interacting witha web browser on device 110, the context of device 110 (or of the user)is “when online.” In response to determining the context of the deviceor of the user, a reminder will be sent to the user to inform the userto email Bob. Additionally another reminder may be provided to the userfor any other task items that are associated with the “when online”context trigger.

As another example, a user command that device 110 receives may be “TextMom when I am talking to my sister Sarah.” The phrase “when I am talkingto my sister Sarah” is presumed to mean that when the user is usingdevice 110 as a phone and a phone call is established with Sarah, areminder will be sent to the user to remind the user to send a text (orSMS) message to the user's mother.

As another example, a user command that device 110 receives may be“Remind me to email Jane Smith when I have a Wi-Fi connection.” Inresponse to device 110 detecting a Wi-Fi signal that does not require apassword or that requires a password accessible to device 110, taskmanager 112 causes a notification to be displayed on a screen of device110, where the notification indicates that Jane is to email Jane Smith.

Non-limiting examples of spatial-oriented context triggers include thespeed at which device 110 is moving (e.g., over 30 mph indicatingdriving, or less than 3 mph indicating walking), a direction (absoluteor relative) at which device 110 is moving, and a set of movements ofdevice 110 (e.g., short vertical movements while moving continuously ina horizontal direction). In other words, device 110 may be configured todetect how device 110 is moving through space.

For example, device 110 (or rather a process executing on device 110)determines, based on detecting changes in its location over a period oftime, that device 110 is moving at 60 mph. Based on this information,device 110 determines that the device's context is “while driving” or“on the road.” Task manager 112 analyzes one or more task items todetermine whether any task items are associated with a “while driving”or “on the road” context trigger. If a task item is associated with a“while driving” or “on the road” context trigger, then an action (e.g.,displaying a notification) associated with the task item is performed.

As another example, device 110 determines, based on detecting changes inits location over a period of time, that device 110 is moving towardshis home over a certain period of time (e.g., 5 minutes). Based on thisinformation, device 110 determines that the context is “on my way home.”Task manager 112 analyzes one or more task items to determine whetherany task items are associated with a “on my way home” context trigger.If a task item is associated with a “on my way home” context trigger,then an action (e.g., displaying a notification) associated with thetask item is performed.

As another example, device 110 includes an accelerator that detectscertain repetitive movements. Device 110 may determine, based on theserepetitive movements over a period of time, that the user of device 110might be running at a slow pace. Based on this determination, device 110determines that the context is “while jogging.” Task manager 112analyzes one or more task items to determine whether any task items areassociated with a “while jogging” or “while walking” context trigger. Ifa task item is associated with a “while jogging” or “while walking”context trigger, then an action (e.g., displaying a notification)associated with the task item is performed.

As another example, device 110 might detect that it has not moved for aperiod of time (e.g., 3 hours). A user of device 110 might be interestedin being alert and non-movement of device 110 might indicate that theuser is asleep. Thus, the user might issue the command, “Alert me if thephone doesn't move for 3 hours.”

In addition to data-oriented and spatial-oriented triggers, other kindsof triggers may be based on any sensor on device 110. Device 110 mayinclude multiple sensors, such as temperature sensors and light sensors.For example, device 110 might include a thermometer for detecting theoutside temperature or an internal temperature of device 110. Thus, auser of device 110 might issue the command, “Remind me to call Haroldwhen it reaches 100 degrees.”

Exogenous Triggers

Another type of triggering criteria that may be associated with a taskitem is exogenous criteria. An “exogenous trigger” is a triggeringcriterion that depends on one or more factors that exist outside andseparate from device 110 and the user of device 110. Such factors may beconsidered “events” that occur with respect to devices other than device110 or with respect to data that is stored on one or more devices otherthan device 110. Non-limiting examples of exogenous triggers includesocial location, social proximity, standing queries, and local events.

An example of a social location trigger is when a friend or associate ofthe user of device 110 arrives or leaves a certain location. Forexample, a user command that initiated the creation of a task item mayhave been “Notify me if Sarah leaves the mall.” Thus, the location ofSarah (or Sarah's mobile device) is an essential factor in setting offthis type of exogenous trigger. Specifically, task manager 112determines the current location of Sarah's device. The current locationof Sarah's device may be provided by a cloud service (e.g., in cloud130) to which both Sarah's device and device 110 are subscribed. Device110 receives, from the cloud service, updates as to the location ofSarah's device. Task manager 112 uses that location information todetermine whether the social location trigger should be activated. Asimilar user command is “Remind me when my daughter gets home.”

An example of a social proximity trigger is when a friend or associateof the user of device 110 is within a certain distance of the user (ordevice 110). For example, a user command that initiated the creation ofa task item may have been “Remind me to call George when he is within100 feet of me.” Thus, the location of George (or George's mobiledevice) is an essential factor in setting off this exogenous trigger.Specifically, task manager 112 or another process executing on device110 compares the current location of device 110 with the currentlocation of George's device to determine the distance that separates thetwo devices. Alternatively, George's device may transmit its location toa cloud service to which both George's device and device 110 aresubscribed. Device 110 receives, from the cloud service, updates as to adistance between George's device and device 110. Task manager 112 usesthat distance information to determine whether the social proximitytrigger should be activated.

An example of a standing query trigger is when a webpage mentions aparticular term or phrase, such as a company name. To detect this, astanding query is generated and issued continuously (e.g., once a day).For example, a user command that initiated the creation of a task itemmay have been “Tell me when cnn.com mentions Berkman Industries.” Taskmanager 112 or another process executing on device 110 issues a searchquery (e.g., to a search engine) and receives results. When task manager112 determines that the results include a webpage from cnn.com thatincludes the name “Berkman Industries,” task manager 112 provides anotification to the user of device 110.

An example of a local event trigger is when a certain local eventoccurs. To detect this, task manager 112 receives data from an onlineservice. Task manager 112 (or a task service in cloud 130) mayperiodically send a request to the online service (via one or morecommunication protocols). Alternatively, task manager 112 may subscribewith the online service to receive information about certain events. Forexample, a user command that initiated the creation of a task item mayhave been “Tell me when Beatles tickets go on sale at Shoreline.” Inresponse, task manager 112, another process executing on device 110, orNLP service 132 sends a subscription request to an online ticket serviceto receive a notification when Beatles tickets for a performance atShoreline Amphitheatre become available for purchase. When task manager112 is determines Beatles tickets are available for purchase, taskmanager 112 provides a notification to the user of device 110.

As another example, a user might be interested in knowing when the surfis up. Thus, the user might issue the command, “Remind me an hour beforethe surf is up.” Task service 112 (or a task service in cloud 130) mightregularly issue a query of a surfing site or might subscribe for alertsfrom the surfing site.

Based on the foregoing, the types and examples of exogenous triggers arevirtually endless. As long as task manager 112 (or a task service incloud 130) can make a determination about an event that occurs separatefrom device 110, that event can be used to trigger the performance of anaction associated with a task item.

III. Consuming Task Items (Active Payloads)

A task item is “consumed” when an action associated with the task itemis performed. Such an action may be a notification that is displayed (orplayed, if the notification is an audio notification) on device 110. Inaddition to or instead of providing a notification to a user of device110, other possible actions include initiating a phone call or a searchquery, sending an HTTP request (that includes a Uniform ResourceLocation (URL)), sending an email or a text (SMS) message, causing anapplication to execute, and causing a purchase to be made on the user'sbehalf. Such actions that can be associated with task items are referredto as “active payloads.” The processing of an active payload causes someaction to be performed, whether by task manager 112 or by anotherprocess, whether local or remote to device 110. In other words, insteadof simply notifying the user of a task associated with a task item, taskmanager 112 (or a service in cloud 130) can automate the action part ofthe task item.

As alluded to above, causing an action to be performed may involve taskmanager 112 causing another application or process to perform theaction. The calling or invoking of the other application (e.g., via anAPI of the other application) may be performed with or without furtherinput, as indicated in the following examples.

The types of “other” applications can vary greatly. Non-limitingexamples of applications that might be available on device 110 include aphone application, an email application, a Web browser, a music playerapplication, a media player application, a music download application,an image processing application, a geopositioning application, acontacts application, an SMS application, a video game application, anda text processing application.

For example, a user of device 110 says aloud, “Remind me to call herback this afternoon.” This voice input is converted into voice data thatdevice 110 sends (along with context data) over network 120 to NLPservice 132. NLP service 132 analyzes the voice data and the contextdata to determine that “her” refers to Marilyn Merlot. NLP service 132determines that “afternoon” is 2 PM (whether based on context data, apre-defined setting, or prior history) and determines a phone number forMarilyn Merlot based on a contacts list (or address book), associatedwith the user, that includes one or more phone numbers for MarilynMerlot. The contacts list may be stored on device 110 or in cloud 130.NLP 132 sends, to task manager 112 (or to a task service in cloud 130),reminder data used to generate a task item. The reminder data includesthe date of “Today”, time of 2 PM, and an instruction to call MarilynMerlot using a particular phone number. When task manager 112 determinesthat the current time is 2 PM, task manager 112 may cause a message tobe displayed that prompts the user to call Marilyn Merlot. The messagemay include a “Later” button and a “Call Now” button. If the userselects the “Later” button, then task manager 112 will send the messageagain later in the afternoon (e.g., in 1 hour). If the user selects the“Call Now” button, then task manager 112 initiates a call to MarilynMerlot. This initiation may involve task manager 112 making an API callto a phone application (not shown) executing on device 110 and passingthe phone number as an argument of the API call. The phone applicationthen uses the phone number to call a device associated with the phonenumber.

As another example, a user of device 110 says aloud, “Text Lindsay thatI love her at 5 o'clock.” This voice input is converted into voice datathat device 110 sends over network 120 to NLP service 132. NLP service132 analyzes the voice data to determine that a cell phone number ofLindsay is necessary and that “5 o'clock” refers to 5 PM of the currentday. Task manager 112 (or a task service in cloud 130) creates a taskitem that includes the following data items: (1) a completion time of 5PM today, (2) an action of sending a text (or SMS) message, (3) a numberof Lindsay's cell phone, and (4) a text string of “I love you” that willbe part of the text message. In response to determining that the currenttime is 5 PM, task manager 112 analyzes the task item to determine theaction that needs to be performed. Task manager 112 then causes a textmessage that includes the text string associated with the task item tobe sent to Lindsay's cell phone. This step may comprise task manager 112invoking an API call of a texting application (not shown) executing ondevice 110, where the text string (“I love you”) is an argument of theAPI call.

As another example, a user of device 110 says aloud, “Show me directionson how to get to Rachel's Restaurant in San Jose when I leave theoffice.” This voice input is converted into voice data that device 110sends over network 120 to NLP service 132. NLP service 132 analyzes thevoice data to determine that a cell phone number of Lindsay is necessaryand that “5 o'clock” refers to 5 PM of the current day. Task manager 112(or a task service in cloud 130) creates a task item that includes thefollowing data items: (1) a location trigger of leaving the user'soffice and (2) an action of displaying instructions (and, optionally, amap) on how to arrive at Rachel's Restaurant from the user's office. Inresponse to determining that the user of device 110 has left his/heroffice, task manager 112 analyzes the task item to determine the actionthat needs to be performed. Task manager 112 then causes (withoutfurther input from the user) a travel directions request to be sent to atravel directions service. The travel directions request includes thename of the restaurant, any address information of the restaurant, orboth. The travel directions service may be hosted on device 110 or onanother device (not shown).

As another example, a user of device 110 says aloud, “Order a cheeseonly pizza at Pizza Heaven in San Jose, home delivered, 30 minutesbefore the Bulls-Pacers game starts.” This voice input is converted intovoice data that device 110 sends over network 120 to NLP service 132.NLP service 132 analyzes the voice data to determine that a Bulls-Pacersgame starts at 6 PM local time; thus, the time trigger is 5:30 PM localtime. NLP service 132 also determines that Pizza Heaven in San Joseallows online ordering. Task manager 112 (or a task service in cloud130) creates a task item that includes the following data items: (1) atime trigger of 5:30 PM and (2) an action of ordering a cheese onlypizza from Pizza Heaven with home delivery as an option. In response todetermining that the current time is 5:30 PM, task manager 112 (or atask service in cloud 13) analyzes the task item to determine the actionthat needs to be performed. Task manager 112 then causes a pizza orderrequest to be sent to Pizza Heaven's online ordering service. The pizzaorder request includes the pizza type of cheese only, the deliveryoption of home delivery, and the user's home address. The pizza orderrequest may be in the form of an API call to the online orderingservice, where arguments of the API call include indications of cheeseonly topping, home delivery, and the user's home address. Alternatively,before causing the pizza order required to be sent, task manager 112 mayformulate a message that is displayed on (or played by) device 110,where the message informs the user about this task. If the user providesaffirmative input, then task manager 112 causes the pizza request orderto be sent. If the user provides negative input, then no pizza requestorder is sent.

As another example, a user of device 110 says aloud, “Play my classicalstation on Pandora at 3 PM tomorrow.” The time of “3 PM tomorrow”coincides with the end of a scheduled exam for the user. This voiceinput is converted into voice data that device 110 sends over network120 to NLP service 132. NLP service 132 analyzes the voice data todetermine a date and time that the intended action is going to beperformed. Task manager 112 (or a task service in cloud 130) creates atask item that includes the following data items: (1) a time trigger of3 PM with a date that identifies the following day and (2) an action ofplaying a classical “station” of the user's Pandora music application,where the classical station was established by the user and associatedwith the user's Pandora account. In response to determining that thecurrent time is 3:00 PM on the proper date, task manager 112 (or a taskservice in cloud 13) analyzes the task item to determine the action thatneeds to be performed. Task manager 112 then causes the Pandora musicapplication (not shown) to begin executing on device 110 and to “play”the user's classical station. Task manager 112 may cause the classicalstation to play by invoking an API call to the Pandora musicapplication, where an argument of the API call includes an indication ofa classical station.

Automated Task Completion

In an embodiment, task manager 112 (or a task service in cloud 130)“marks” the task item as complete in response to detecting that a taskitem is consumed. In other words, a task item may be associated with acomplete or an incomplete status. Task manager 112 may provide aninterface for a user to view task items managed by task manager 112 anddetermine whether a task item is complete or not. Task manager 112 mayprovide an option for a user of device 110 to view all completed taskitems. The completed task items may be ordered based on when the taskitems were created, consumed (or completed), or some other criteria.

Additionally or alternatively, task items that are consumed are deletedfrom storage. For example, task manager 112 deletes, from storage ondevice 110, any task items that have been consumed. The deletion of atask item may occur a certain period of time (e.g., 1 month) after thecorresponding task has been completed to allow a user of device 110 toreview recently-consumed task items. If a task service in cloud 130manages task items that are stored in cloud 130, then that task servicemay delete consumed task items.

Delayed Task Interpretation

In an embodiment, when a task item is created, only some details of thecorresponding task may be known and stored in association with the taskitem. Other details regarding the description, address (if any),trigger, and/or action may be determined later, whether automatically orvia a manual process.

For example, device 110 sends, to NLP service 132, voice data thatreflects a user command to “Call Sarah at 5.” NLP service 132 determinesthat 5 PM of the current day is a time trigger and causes task manager112 (or a task service in cloud 130) to create a task item with thattime trigger. However, an action item associated with the task item is“Call Sarah” without any indication of a phone number. NLP service 132has not yet determined who Sarah is and, thus, what phone number to useto call her. Instead, those details are determined later; for example,when the current time is 5 PM and the action is triggered or sometimebefore the trigger activates. At 5 PM, task manager 112 sends the actionitem “Call Sarah” (whether in text form or audio form) to NLP service132 or another service to identify information about a particular Sarah(if there are many) and to determine a phone number for Sarah. When aphone number for Sarah is determined, task manager 112 (or anotherprocess) causes a phone application on device 110 to initiate a callusing the phone number. In this example, the disambiguation of (a) theidentity of an individual and (b) a phone number for that individual isdelayed until after the task item is generated.

As another example, device 110 sends, to NLP service 132, voice datathat reflects a user command to “Check the weather in San Jose tomorrowmorning.” NLP service 132 determines that 7 AM of the next day is a timetrigger and causes task manager 112 (or a task service in cloud 130) tocreate a task item with that time trigger. However, an action itemassociated with the task item is “Check the weather in San Jose” withoutany indication of how to perform the action. NLP service 132 has not yetinterpreted that portion of the user command to determine how theweather in San Jose is to be checked. Instead, those details aredetermined later; for example, when the current time is 7 AM of the nextday and the action is triggered or sometime before the triggeractivates. At 7 AM of the next day, task manager 112 sends the actionitem “Check the weather in San Jose” (whether in text form or audioform) to NLP service 132 or another service to identify how the weatherin San Jose is to be checked. In response, NLP service 132 or anotherservice retrieves information about the weather in San Jose and providesthat information to device 110 to be displayed. In this example, thedetermination of how the action is to be performed is delayed untilafter the task item is generated.

Response to Alert

As alluded to previously, for a task item that is associated with anaction that is more than a mere notification, instead of performing theaction, a user of device 110 is first alerted of a task and the user isallowed to respond with an affirmative or negative response. Forexample, an action of a task item is to email Jane Smith about ProjectKnuckles. Task manager 112 causes, to be displayed on device 110, amessage that indicates that the user of device 110 is suppose to emailJane Smith. The user may press a physical or graphical button thatindicates an affirmative response. Alternatively, the user may speak thecommand, “Do it” or “Yes” indicating an affirmative response. Inresponse to the input (whether via a touch screen of device 110, akeyboard selection, or voice input), task manager 112 causes an emailapplication on device 110 to compose an email message addressed to JaneSmith with a subject line that refers to Project Knuckles.Alternatively, the user may decide to be reminded later of the task toemail Jane Smith. Thus, in response to the notification, the userprovides input (via device 110) that indicates that s/he would like toemail Jane Smith some time later, such as in one hour or the next day.Such input may be the user saying “Remind me later” or simply “later.”

In an embodiment, when the action is to respond to an act ofcommunication such as an email message, task manager 112 stores thecontext of the communication at the time of task creation and retrievesthe context at the time of performing the action. The context ofcommunication might be, in various embodiments, a Universal ResourceIdentifier or other reference to the context or a copy of the data ofthe context. For example, task manager 112 stores a reference to or copyof the email message that is to be replied to. When the action isperformed, the contents of the email message can be recreated just as ifthe user had performed a reply when initially reading it. Other examplesof context data that can be stored and retrieved in this manner includewithout limitation text messages, documents, web pages, voicemailmessages, photographs, audio recordings, and videos.

As another example, an action of a task item is to call George Burt. Inresponse to determining to trigger the action to call, task manager 112provides an indication that a reminder is available for a user of device110. The indication may be device 110 buzzing/shaking, generating anaudible noise, and/or displaying a notification message. Without holdingdevice 110, the user says aloud, “Read it.” In response to task manager112 (or another process) processing this input, device 110 plays anaudible version of the following statement: “Reminder . . . call GeorgeBurt.” The audible version may be based on a playback of the originalinput from the user or may reflect a computer-generated voice. If theuser decides to call George Burt, then the user may simply say, “Okay”or “Do it,” which causes a phone application on device 110 to callGeorge Burt. If the user decides not to call George Burt, then the usermay say, “Ignore” or “remind me later.”

IV. Organizing Task Items Using Lists

According to an embodiment of the invention, a task item may beassociated with one or more lists. A list is a set of one or more taskitems that are associated with (or belong to) the same category. Listsare ways that a user of device 110 can view task items in an organizedway. The different lists allow the user to intelligently and intuitivelybrowse the tasks that s/he would like to perform (or have performed onhis/her behalf). FIGS. 6-14 depict views of various types of lists,according to an embodiment of the invention.

When a new task item is created, task manager 112 (or a service in cloud130) identifies one or more attributes associated with the new task itemand assigns the new task item to one or more lists. For example, if thenew task item includes the action “to call,” then task manager 112 (orother process) adds the new task item to a To Call list. Similarly, ifthe new task item includes a certain context and a particular location,then task manager 112 might identify the context and/or the particularlocation and add the new task item to a location list and/or a contextlist. Alternatively, a user might manually identify one or more of thelists, which are described in detail below, to which a new task item isto be added.

All Lists View

FIG. 5A depicts an All Lists view 500 that device 110 might display,according to an embodiment of the invention. All List view 500 does notcontain information about any specific task items. Instead, All Listsview 500 includes references to multiple lists maintained by taskmanager 112 (or a task service in cloud 130): a Today list 510, an AllTo Do list 520, a Nearby list 530, an In Car list 540, a To Call list550, a To Email list 560, a Groceries list 570, a To Buy list 580, and aCompleted list 590. As noted previously, a task item may be associatedwith (or belong to) multiple lists. For example, a task item whosedescription is to buy milk and whose time trigger is today may belong toToday list 510, All To Do list 520, Groceries list 570, and To Buy list580.

Lists may be characterized as one of three types: built-in or predefinedlist, smart list, or custom list. Today list 510, All To Do list 520,and Completed list 590 are examples of built-in or pre-defined lists.

Smart lists are based on different characteristics or attributes that atask item might have, such as an action (e.g., call, email, text,alert), a location, and/or a context in which the action is to beperformed. Examples of smart lists include By Action lists, By Locationlists, and By Context lists. In Car list 540, To Call list 550, and ToEmail list 560 are examples of By Action lists. Other examples of ByActions lists might include a To Text list, a To Lookup list, and a ToVisit list.

Examples of custom lists include lists that are based on categoriesidentified by NLP service 132 and lists that are created by a user.Groceries list 570 and To Buy list 580 are examples of custom lists.Another example of a custom list is a wine list (not shown) thatincludes a list of the user's favorite wines.

Returning to the lists depicted in FIG. 5A, task items that belong toToday list 510 are associated with a triggering criterion that indicatesa time during the current day that the corresponding task must or shouldbe performed. All task items belong to All To Do list 520. Task itemsthat belong to Nearby list 530 are associated with locations that areconsidered to be within a certain distance (e.g., 1 mile) from thecurrent location of device 110. Task items that belong to In Car list540 are associated with tasks that are to be performed in a car or whiletraveling. Task items that belong to To Call list 550 are associatedwith the action to call a person or entity. Task items that belong to ToEmail list 560 are associated with the action to email a person orentity. Task items that belong to Groceries list 570 are associated withgrocery items (e.g., milk, eggs, fruit) to purchase. Task items thatbelong to To Buy list 580 are associated with items to purchase, such asclothing, books, songs, or groceries. Task items that belong toCompleted list 590 are considered completed, which may indicate that thecorresponding tasks have been performed or at least that an action(e.g., an alert or notification) associated with each task item has beenperformed.

All Lists view 500 also includes a “+” image that when selected, allowsa user of device 110 to create another custom list so that currentand/or future task items can be added thereto.

FIG. 5B depicts some of the lists depicted in FIG. 5A, but with a searchfield 502 to allow a user of device 110 to search for a specific taskitem. A task item may be searched for based on, for example, the taskitem's associated creation date, completion date (if known), completionstatus, context trigger (if any), location (if any), and/or action type(e.g., notify only, call, email, or buy).

Today List

FIG. 6 depicts a view 600 of a Today list that device 110 displays, forexample, in response to user selection of Today list 510. View 600includes a list of tasks that are divided into two sections: a section610 for task items that are associated with a specific time and asection 620 for task items that are not associated with a specific time.Each of the task items in section 610 is associated with a travel timereminder. The third task item in section 610 and the second throughfourth task items in section 620 are associated with actions that aremore than mere reminders or alerts.

For example, the third task item in section 610 is to “pick up Chloe” at5:00 PM. The icon to the right of that description is an image of acompass, indicating that the action associated with this task item is togenerate travel directions to help guide the user of device 110 to theintended destination, which is Pinewood School in this example.

As another example, the second task item in section 620 is to “call JohnAppleseed.” The icon to the right of that description is an image of aphone, indicating that the action associated with this task item is tocall John Appleseed. The image adjacent to the phone image is of a car,indicating that the user of device 110 is to call John Appleseed whenthe user is in a car or while the user is traveling.

As another example, the last task item in section 620 is to “reply toPablo Marc.” The icon to the right of that description is an image of anenvelope, indicating that the action associated with this task item isto send an email to Pablo Marc. View 600 also indicates that this taskitem is overdue, or rather, that the originally-scheduled time to emailPablo Marc has passed.

Single Task Item View

FIG. 7 depicts a view 700 that device 110 displays and that includesdetails about a particular task item. View 700 may have been generatedbased on a user selection of the second task item in section 620 in view600 of FIG. 6. The displayed task item contains four data items: adescription item 710, an action item 720, a reminder item 730, and alist assignment item 740.

Description item 710 contains a high-level description of the task(“Call John Appleseed”) and includes details about the subject matter(“Discuss the almond deal”). Selection of description item 710 may allowa user of device 110 to edit the description.

Action item 720 contains a description of the action (“Call”) andincludes which phone (“mobile”) of John Appleseed to use. Selection ofaction item 720 may allow the user of device 110 to view the phonenumber associated with John Appleseed and/or provide other contactoptions, such as another phone number associated with John Appleseed, anemail address of John Appleseed, etc. Furthermore, selection of thephone icon in action item 720 may cause task manager 112 to initiate acall phone to John Appleseed right then instead of waiting for the oneor more triggering criteria associated with the task item to besatisfied.

Reminder item 730 indicates the type of trigger (“when in car”) that,when detected, will cause the action to be performed, or at least analert about the task. Selection of reminder item 730 may allow a user tochange the type of reminder.

List assignment item 740 indicates the list to which the task itembelongs, which is the “Nut to Crack Project” list in this example. Thislist is an example of a customized list. Selection of list assignmentitem 740 may cause device 110 to display multiple task items that belongto the “Nut to Crack Project” list.

All to do List

FIG. 8 depicts a view 800 of an All To Do list that device 110 displaysand that includes information about multiple task items. In thisexample, the multiple task items are ordered by date. View 800 may havebeen generated based on a user selection of All To Do list 820 in view800 of FIG. 8A. View 800 is divided into two sections: section 810 thatcontains task items (or references thereto) to be completed on one dayand section 820 that contains task items to be completed on thefollowing day.

Some of the task items referenced in view 800 have been completed. Suchcompleted task items are shown with a lighter gray image to the left ofthe corresponding description. Task items that have been completed maybe distinguished from not-yet-completed task items by other techniques,such as check marks.

In the example depicted in FIG. 8, the task items are organized by thedate on which the corresponding tasks should be performed (or “duedate”). However, the task items referenced in view 800 may be organizedby the date on which a user of device 110 is to be alerted or remindedof the corresponding tasks (“alert date”), the date on which the taskitems were created (“created date”), the date on which the task itemswere modified (“modified date”), or the date on which the correspondingtasks were performed (“completed date”).

Nearby List

FIG. 9 depicts a view 900 of a “Nearby” list that device 110 displays.View 900 may have been generated based on a user selection of Nearbylist 830 in view 800 of FIG. 8A. View 900 contains information aboutmultiple locations that are ordered based on distance from device 110'scurrent location. The location indicated at the top of the list (“Home”)is closest to the current location of device 110 while the locationindicated at the bottom of the list (“Pinewood School”) is furthest fromthe current location of device 110.

Each location indicated in view 900 is associated with a differentlocation list. Each location list may be associated with one or moretask items. For example, the “Home” location may be associated with fourtask items (which may be displayed on user selected of the “Home”location) while the “Atherton Dry Cleaning” location may be associatedwith just one task item.

Because the locations indicated in view 900 are ordered based ondistance from the current location of device 110, when the currentlocation of device 110 changes, the location indicators may bere-ordered, some may be removed from view 900, and others not currentlydisplayed in view 900 may appear in view 900. For example, if device 110is currently located in a store that is next to the Whole Foods storeidentified by the second location indicated in view 900, then, if device110 displays view 900, that Whole Foods location indicator will be atthe top of the list.

As indicated above, view 900 includes a “Home” location and a “Work”location. The association of a location labeled “Home” (or “Work”) witha particular address may be made in numerous ways. For example, manymobile devices store profile information about a user of the mobiledevice. This information is referred to as a “me card.” A me cardtypically stores a user's home address and the user's work address.Thus, task manager 112 (or another process) analyzes the me card that isstored on device 110 to determine a home address and a work address (ifany) of the user.

In an embodiment, a radius is associated with a particular location andany task items that are associated with a location that is within thedistance indicated by the radius is considered to be associated with theparticular location. For example, a radius associated with a home of auser of device 110 is 2 miles. If a task item is associated with a parkand the park is within 2 miles from the home, then the task item isassociated with a “home” list, along with other task items that areassociated with the home.

Location List View

As noted previously, a location list is an example of a smart list. Inan embodiment, any task item that is associated with a location (e.g.,as part of the one or more triggering criteria) is automaticallyassociated with a location list that is associated with the samelocation as the location of the task item. Task manager 112 (or a taskservice in cloud 130) may maintain multiple location lists.

FIG. 10A depicts a Location List view 1000 that device 110 displays.Location list view 1000 may have been generated based on a userselection the “Home” location indicator in Nearby view 900 of FIG. 9.Location list view 1000 contains six task items. The bell image adjacentto each of the first four task items indicates that a reminder (oralert) for those task items will be generated when device 110 is at ornear the user's home or at least sometime on a specified date. Areminder or alert will not be generated for the last two task items.

Location List view 1000 also includes a map icon 1002 which, whenselected, causes task manager 112 to communicate with a map applicationthat generates a map of the location associated with the map icon. Inthis example, a map of the user's home would be generated.

FIG. 10B depicts a Location List view 1050 that device 110 displays.Location List view 1050 may have been generated based on a userselection the “Whole Foods” location indicator in Nearby view 900 ofFIG. 9. Location List view 1050 contains six data items, each of whichmay or may not be a task item. Instead, each data item in Location Listview 1050 simply identifies a grocery item to purchase at a Whole Foodsgrocery store. None of the grocery items are associated with a reminder(although they could be) or a completion date (although they could be).

The grocery items identified in Location List view 1050 was associatedwith the Whole Foods grocery list in response to input from a user ofdevice 110. For example, a user spoke the following command: “Add almondmilk to my grocery list” or “Remember to pick up almond milk at WholeFoods near my house.” Device 110 transmits voice data that reflects thiscommand to NLP service 132. NLP service 132 determines, based on thevoice data, that the user intends to purchase almond milk. NLP service132 may cause task manager 112 to (a) create a task item for the task ofpurchasing almond milk and add the task item to the Whole Foods list or(b) simply add “almond milk” to the Whole Foods list.

Location List view 1050 also includes a map icon 1052 which, whenselected, causes task manager 112 to communicate with a map applicationthat generates a map of the location associated with the map icon. Inthis example, a map of the Whole Foods store identified by the displayedaddress would be generated.

Smart Lists

As noted previously, By Location lists, By Action lists, and By Contextlists are examples of smart lists. FIG. 11A depicts a view 1100 of a ByContext list; specifically, an In Car list. FIG. 11B and FIG. 11D depictviews of different By Action lists; specifically, a To Call list and aTo Email list.

View 1100 contains task items that are associated with tasks that are tobe performed in a specific context, i.e., the “In Car” context. The taskitems in the In Car list may be associated with different actions, suchas calling and getting directions.

In contrast, view 1110, depicted in FIG. 11B, contains task items thatare associated with the same action, which, in this example, is to calla person or entity. The first three task items in view 1110 have a phoneicon, indicating that a phone number for the person indicated in thecorresponding task is known to task manager 112. However, the last taskitem in view 1110 is not associated with a phone icon, indicating that aphone number for “Bob” is not positively known to task manager 112,probably because many contacts in the user's contact list may have thename of Bob. Selection of the “call Bob” task item in view 1110 causesdevice 110 to display a view 1120 depicted in FIG. 11C.

View 1120 indicates two data items that are contained in (or associatedwith) the “call Bob” task item: a description item and an action item.The action item indicates that multiple contacts are known as “Bob.” Asa result, the action item includes a call button that is disabled,whereas the call buttons associated with the other task items in view1110 are not disabled. Selection of the action item may initiate aprocess for disambiguating the identity of “Bob.” For example, selectionof the action item may cause task manager 112 to display a list ofnames, each of which have the name of Bob or Robert. In this way, thedisambiguation of an identity or of a phone number may occur much laterthan the creation of the corresponding task item.

View 1130, depicted in FIG. 11D, includes six task items, each of whichincludes an action to email. The active payload arguments of a To Emailtask item include a “To” or email address and, optionally, a subject forthe subject line of the email.

In an embodiment, an “email” task item is created from an emailapplication that is separate from task manager 112. The emailapplication may invoke an API call of task manager 112 to create a taskitem whose action is to email, where the action includes an activepayload that includes an email address and a subject.

Custom Lists

As noted previously, custom lists are one of the three main types oflists, including built-in lists and smart lists. Examples of customlists indicated above include Grocery list 570 and To Buy list 580(referenced in FIG. 5A). FIG. 12 depicts a view 1200 that might begenerated in response to user selection of Grocery list 570. View 1200includes six data items, each referring a different grocery item topurchase. Each of these data items may be task items that only have adescription. The data items may have been associated with the grocerylist based on input from NLP service 132. For example, NLP servicereceives, from device 110, voice data that reflects the user command to“pick up fresh bread from the store.” NLP service 132 determines thatthe user of device 110 intends to purchase fresh bread from a grocerystore and associates “fresh bread” with a grocery category. In response,NLP service 132 sends, to task manager 112, a create task item commandto create a task item that includes the description “fresh bread” andthat is associated with the grocery category. In response, task manager112 creates a task item and associates the task item with a grocery listthat task manager 112 maintains.

FIG. 13 depicts a view 1300 of another type of custom list: auser-defined list. This user-defined list is entitled, “Nut to CrackProject,” and contains three task items, the first of which isassociated with an action (i.e., call) and a context trigger (e.g. “incar” or “while driving”). A user of device 110 may “manually” associatea task item with a user-defined list. For example, after task manager112 creates a task item, the user selects the task item and, via one ormore selectable (e.g., menu) options displayed on device 110, selects aparticular user-defined list, which causes task manager 112 to associatethe task item with the particular user-defined list.

Alternatively, NLP service 132 may determine, based on input data(whether voice or text) received from device 110, a specific list toassociate with a task item. For example, voice data may reflect a usercommand to “I need to write a proposal for the Nut to Crack Project.”NLP service 132 determines that “write a proposal” is the task and that“Nut to Crack Project” is the name of a list, which task manager 112 mayor may not have yet created. NLP service 132 then sends, to task manager112, the description (“write proposal”) and the name of a possible listto which the to-be-created task item may be added (“Nut to CrackProject”). Task manager 112 determines whether there is a list that hasthe same or similar name as “Nut to Crack Project.” If so, then taskmanager 112 creates a new task item and associates the task item withthat list. If not, then task manager 112 creates a new list with thatname, creates a new task item, and associates that task item with thenew list.

Lists and Notes

As noted previously, a list may contain items that are not tasks. Such“non-task” are referred to as “notes” that consist only of adescription. FIG. 14 depicts a view 1400 of a Favorite Wines list, whichcontains six notes, each referring to a different wine.

Also as noted previously, NLP service 132 may be configured to recognizelist names so that task manager 112 can easily assign tasks and notes tothe appropriate list(s).

Calendar Events

In an embodiment, calendar events created in the context of a calendarapplication are used to create task items that are managed by taskmanager 112. The calendar application may be part of task manager 112 ormay be separately executing applications. For example, the calendarapplication might be configured to send newly-created calendar events totask manager 112, e.g., via one or more API calls that cause taskmanager 112 to create a task item based on the details of a calendarevent, such as a description, a date, a location (if any), a duration(if any), and a reminder (if any). Alternatively, task manager 112 mightprovide a calendar service that allows a user to view a calendar andcreate events that are associated with a specific date and time or setof dates. Upon creation of events, task manager 112 also creates taskitems for the events.

FIG. 15 depicts a view 1500 of a task item that was generated based on acalendar event. The task item includes four data items: a description(“lunch with Elizabeth Reid”), a begin time (“12:00 PM Today”), aduration (“1 hour”), and a reminder (“10 minutes before”). Selection ofany of the four data items may allow a user of device 110 to edit thecorresponding data items. In an embodiment, if a change is made to atask item that was generated based on a calendar event, then that changeis “pushed” to the calendar event that is managed by a calendarapplication.

In either scenario, if a calendar event that is created and maintainedby the calendar service is associated with a location, then a task itemthat is generated based on the calendar event might also be associatedwith the location. In that case, task manager 112 might automaticallyassociate the task item with a location list, such as the location listin view 1000 of FIG. 10A.

Combinations

While the foregoing description includes four main approaches(generating task items, organizing task items, triggering notifications,and consuming task items), each of these approaches may be implementedindividually or may be used together, as noted in many of the examples.For example, natural language processing may be used to generate a taskitem, but none of the approaches described herein for processing thetask item (i.e., organizing the task item, triggering a notification,and consuming the task item) are used. As another example, naturallanguage processing may be used to generate a task item and an approachfor organizing the task item as described herein may be used, but noneof the approaches for triggering a notification or consuming the taskitem described herein are used. As another example, none of theapproaches for generating and organizing task items and triggering anotification is used, but the approach for consuming the task item asdescribed herein is used.

Hardware Overview

According to one embodiment, the techniques described herein areimplemented by one or more special-purpose computing devices. Thespecial-purpose computing devices may be hard-wired to perform thetechniques, or may include digital electronic devices such as one ormore application-specific integrated circuits (ASICs) or fieldprogrammable gate arrays (FPGAs) that are persistently programmed toperform the techniques, or may include one or more general purposehardware processors programmed to perform the techniques pursuant toprogram instructions in firmware, memory, other storage, or acombination. Such special-purpose computing devices may also combinecustom hard-wired logic, ASICs, or FPGAs with custom programming toaccomplish the techniques. The special-purpose computing devices may bedesktop computer systems, portable computer systems, handheld devices,networking devices or any other device that incorporates hard-wiredand/or program logic to implement the techniques.

For example, FIG. 16 is a block diagram that illustrates a computersystem 1600 upon which an embodiment of the invention may beimplemented. Computer system 1600 includes a bus 1602 or othercommunication mechanism for communicating information, and a hardwareprocessor 1604 coupled with bus 1602 for processing information.Hardware processor 1604 may be, for example, a general purposemicroprocessor.

Computer system 1600 also includes a main memory 1606, such as a randomaccess memory (RAM) or other dynamic storage device, coupled to bus 1602for storing information and instructions to be executed by processor1604. Main memory 1606 also may be used for storing temporary variablesor other intermediate information during execution of instructions to beexecuted by processor 1604. Such instructions, when stored innon-transitory storage media accessible to processor 1604, rendercomputer system 1600 into a special-purpose machine that is customizedto perform the operations specified in the instructions.

Computer system 1600 further includes a read only memory (ROM) 1608 orother static storage device coupled to bus 1602 for storing staticinformation and instructions for processor 1604. A storage device 1610,such as a magnetic disk or optical disk, is provided and coupled to bus1602 for storing information and instructions.

Computer system 1600 may be coupled via bus 1602 to a display 1612, suchas a cathode ray tube (CRT), for displaying information to a computeruser. An input device 1614, including alphanumeric and other keys, iscoupled to bus 1602 for communicating information and command selectionsto processor 1604. Another type of user input device is cursor control1616, such as a mouse, a trackball, or cursor direction keys forcommunicating direction information and command selections to processor1604 and for controlling cursor movement on display 1612. This inputdevice typically has two degrees of freedom in two axes, a first axis(e.g., x) and a second axis (e.g., y), that allows the device to specifypositions in a plane.

Computer system 1600 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 1600 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 1600 in response to processor 1604 executing one or moresequences of one or more instructions contained in main memory 1606.Such instructions may be read into main memory 1606 from another storagemedium, such as storage device 1610. Execution of the sequences ofinstructions contained in main memory 1606 causes processor 1604 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any non-transitorymedia that store data and/or instructions that cause a machine tooperation in a specific fashion. Such storage media may comprisenon-volatile media and/or volatile media. Non-volatile media includes,for example, optical or magnetic disks, such as storage device 1610.Volatile media includes dynamic memory, such as main memory 1606. Commonforms of storage media include, for example, a floppy disk, a flexibledisk, hard disk, solid state drive, magnetic tape, or any other magneticdata storage medium, a CD-ROM, any other optical data storage medium,any physical medium with patterns of holes, a RAM, a PROM, and EPROM, aFLASH-EPROM, NVRAM, any other memory chip or cartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire and fiber optics, including thewires that comprise bus 1602. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 1604 for execution. Forexample, the instructions may initially be carried on a magnetic disk orsolid state drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 1600 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 1602. Bus 1602 carries the data tomain memory 1606, from which processor 1604 retrieves and executes theinstructions. The instructions received by main memory 1606 mayoptionally be stored on storage device 1610 either before or afterexecution by processor 1604.

Computer system 1600 also includes a communication interface 1618coupled to bus 1602. Communication interface 1618 provides a two-waydata communication coupling to a network link 1620 that is connected toa local network 1622. For example, communication interface 1618 may bean integrated services digital network (ISDN) card, cable modem,satellite modem, or a modem to provide a data communication connectionto a corresponding type of telephone line. As another example,communication interface 1618 may be a local area network (LAN) card toprovide a data communication connection to a compatible LAN. Wirelesslinks may also be implemented. In any such implementation, communicationinterface 1618 sends and receives electrical, electromagnetic or opticalsignals that carry digital data streams representing various types ofinformation.

Network link 1620 typically provides data communication through one ormore networks to other data devices. For example, network link 1620 mayprovide a connection through local network 1622 to a host computer 1624or to data equipment operated by an Internet Service Provider (ISP)1626. ISP 1626 in turn provides data communication services through theworld wide packet data communication network now commonly referred to asthe “Internet” 1628. Local network 1622 and Internet 1628 both useelectrical, electromagnetic or optical signals that carry digital datastreams. The signals through the various networks and the signals onnetwork link 1620 and through communication interface 1618, which carrythe digital data to and from computer system 1600, are example forms oftransmission media.

Computer system 1600 can send messages and receive data, includingprogram code, through the network(s), network link 1620 andcommunication interface 1618. In the Internet example, a server 1630might transmit a requested code for an application program throughInternet 1628, ISP 1626, local network 1622 and communication interface1618.

The received code may be executed by processor 1604 as it is received,and/or stored in storage device 1610, or other non-volatile storage forlater execution.

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense. The sole and exclusive indicator of the scope of the invention,and what is intended by the applicants to be the scope of the invention,is the literal and equivalent scope of the set of claims that issue fromthis application, in the specific form in which such claims issue,including any subsequent correction.

What is claimed is:
 1. A method performed by an electronic device withone or more processors and memory, the method comprising: receiving anatural language input specifying a task to be performed; generating,based on the natural language input, a first task item for the taskincluding a first action to be performed when a respective user is usinga first mode of transportation, wherein the first action is a firstprogrammed function of the electronic device; obtaining, by the one ormore processors, context data associated with the natural languageinput, wherein the context data includes data displayed or processed bythe electronic device within a threshold duration of receiving thenatural language input; determining, by the one or more processors,based on the natural language input and the context data, an attributeassociated with the first action, the attribute not explicitly specifiedin the natural language input; after generating the first task item:storing, in the memory, a plurality of task items including the firsttask item and a second task item, wherein: the second task item isgenerated based on input from a user of the electronic device; and thesecond task item includes a second action that is of a different typethan the first action, and is to be performed when the respective useris using a second mode of transportation different from the first modeof transportation, wherein the second action is a second programmedfunction of the electronic device; and after storing the plurality oftask items: establishing a wireless communication connection;determining, based on the established wireless communication connection,a respective mode of transportation currently being used by therespective user; in accordance with a determination that the respectiveuser is using the first mode of transportation, causing the first actionincluded in the first task item to be performed by the electronic devicebased on the determined attribute associated with the first action; andin accordance with a determination that the respective user is using thesecond mode of transportation, causing the second action included in thesecond task item to be performed by the electronic device.
 2. The methodof claim 1, wherein the first action is causing an email or a textmessage to be sent over a network to a recipient that is associated withthe first task item.
 3. The method of claim 1, wherein the first actionis initiating a phone call to a device associated with a phone numberthat is associated with the first task item.
 4. The method of claim 1,wherein the first action is initiating a search based on a search queryor URL that is associated with the first task item.
 5. The method ofclaim 1, wherein causing the first action to be performed comprises afirst process causing a second process to perform the first action. 6.The method of claim 5, wherein the first process causing the secondprocess to perform the first action comprises the first process invokingan API call of the second process, wherein the API call comprises thedetermined attribute associated with the first action.
 7. The method ofclaim 1, wherein the natural language input comprises voice input thatwas received at the electronic device.
 8. The method of claim 1, whereindetermining the respective mode of transportation comprises determiningthe respective mode of transportation currently being used by therespective user based on movement of a device associated with therespective user or a change in location over a predefined period of timeof the device associated with the respective user.
 9. The method ofclaim 1, wherein the first action is causing a map application todisplay a map of an area that includes a location associated with thefirst task item.
 10. The method of claim 1, wherein: the plurality oftask items are stored on a device associated with the respective user;and a task manager being executed on the device associated with therespective user causes a first application separate from the taskmanager to perform the first action included in the first task item anda second application separate from the task manager to perform thesecond action included in the second task item.
 11. The method of claim10, wherein the task manager causes the first application separate fromthe task manager to perform the first action by invoking an API call ofthe first application.
 12. A non-transitory computer readable storagemedium storing one or more programs, which, when executed by one or moreprocessors of an electronic device, cause the electronic device to:receive a natural language input specifying a task to be performed;generate, based on the natural language input, a first task item for thetask including a first action to be performed when a respective user isusing a first mode of transportation, wherein the first action is afirst programmed function of the electronic device; obtain, by the oneor more processors, context data associated with the natural languageinput, wherein the context data includes data displayed or processed bythe electronic device within a threshold duration of receiving thenatural language input; determine, by the one or more processors, basedon the natural language input and the context data, an attributeassociated with the first action, the attribute not explicitly specifiedin the natural language input; after generating the first task item:store, in the memory, a plurality of task items including the first taskitem and a second task item, wherein: the second task item is generatedbased on input from a user of the electronic device; and the second taskitem includes a second action that is of a different type than the firstaction, and is to be performed when the respective user is using asecond mode of transportation different from the first mode oftransportation, wherein the second action is a second programmedfunction of the electronic device; and after storing the plurality oftask items: establish a wireless communication connection; determine,based on the established wireless communication connection, a respectivemode of transportation currently being used by the respective user; inaccordance with a determination that the respective user is using thefirst mode of transportation, cause the first action included in thefirst task item to be performed by the electronic device based on thedetermined attribute associated with the first action; and in accordancewith a determination that the respective user is using the second modeof transportation, cause the second action included in the second taskitem to be performed by the electronic device.
 13. The non-transitorycomputer readable storage medium of claim 12, wherein the first actionis causing an email or a text message to be sent over a network to arecipient that is associated with the first task item.
 14. Thenon-transitory computer readable storage medium of claim 12, wherein thefirst action is initiating a phone call to a device associated with aphone number that is associated with the first task item.
 15. Thenon-transitory computer readable storage medium of claim 12, wherein thefirst action is initiating a search based on a search query or URL thatis associated with the first task item.
 16. The non-transitory computerreadable storage medium of claim 12, wherein causing the first action tobe performed comprises a first process causing a second process toperform the first action.
 17. The non-transitory computer readablestorage medium of claim 16, wherein the first process causing the secondprocess to perform the first action comprises the first process invokingan API call of the second process, wherein the API call comprises thedetermined attribute associated with the first action.
 18. Thenon-transitory computer readable storage medium of claim 12, wherein thenatural language input comprises voice input that was received at theelectronic device.
 19. The non-transitory computer readable storagemedium of claim 12, wherein determining the respective mode oftransportation comprises determining the respective mode oftransportation currently being used by the respective user based atleast on movement of a device associated with the respective user or achange in location over a predefined period of time of the deviceassociated with the respective user.
 20. The non-transitory computerreadable storage medium of claim 12, wherein the first action is causinga map application to display a map of an area that includes a locationassociated with the first task item.
 21. The non-transitory computerreadable storage medium of claim 12, wherein: the plurality of taskitems are stored on a device associated with the respective user; and atask manager being executed on the device associated with the respectiveuser causes a first application separate from the task manager toperform the first action included in the first task item and a secondapplication separate from the task manager to perform the second actionincluded in the second task item.
 22. The non-transitory computerreadable storage medium of claim 21, wherein the task manager causes thefirst application separate from the task manager to perform the firstaction by invoking an API call of the first application.